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DETAILED ACTION 



1 . This communication is responsive to the response, filed 10/08/04. 

2. Claims 1-13, 15-28, and 30-35 are pending in this application. In communication, claims 
1, 6, 7, 9, 10, 12, 17, 22, 26, 27, 28, 30, and 31 are independent claims. This action is made final. 

3. The text of those sections of Title 35, U.S. Code not included in this action can be found 
in a prior office action. 

Claim Rejections - 35 USC §102 

4. Claims 1, 4, 5-7, 9, 10, 12, 15, 16, 22, 24, 25, 26-28, and 30 are rejected under 35 
U.S.C. 102(e) as being anticipate by Kuzunuki et al. (U.S. Patent No. 5,917,490). 

As to claim 1, Kuzunuki teaches an information input/output system to be used for user 
operations relating to an object in an information space realized by expanding a digital space of a 
computer into the real world, said system comprising: 

one or more than one operation surfaces arranged in said information space (display 101, 
col. 6 lines 14-32, block 400, col. 6 lines 44-58, and fig. 1); 

a display means for displaying an image on said operation surfaces (e.g., col. 6 lines 14- 
32, projectors 105 and 107, col. 7 lines 11-29, and fig. 2); 

an imaging means for picking up an image of said operation surfaces (e.g., col. 2 lines 
41-51, overhead camera 300 to pick up actual images, col. 6 lines 1-12, and figs. 1-2); 

a stationary environment type computer arranged in said information space (e.g., col. 6 
lines 1-32, lines 44-58, and figs. 1-2); and 
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one or more than one physical objects mounted on said operation surfaces and containing 
a visually identifiable visual marker on the surface thereof (a barcode is assigned to an objects, 
col. 21 lines 20-26, lines 42-58); 

said environmental type computer being adapted to execute (a program which is executed 
by the information processing unit, col. 19 lines 41-65); 

a processing operation of recognizing the identification information and the position 
information of each of said physical objects in said visual marker (e.g., col. 21 lines 20-58); 

a processing operation of recognizing the digital object dropped to a site on the surface of 
each of said physical objects (e.g., col. 7 lines 46-65 and col. 14 lines 6-9); and 

a processing operation of forming link information for linking the digital object to the 
dropped site on the surface for each of said physical objects (e.g., col. 3 lines 44-52, link between 
physical objects and electronically stored information, col. 19 lines 36-54). 

As to claim 4, Kuzunuki teaches the information input/output system according to claim 
1, wherein 

said imaging means can identify the position indicated by an optical pointer for indicating 
a specific position by irradiating a beam of light with a predetermined wavelength (using infrared 
light, col. 21 lines 20-26); and 

the user is allowed to indicate a position in said information space by means of 
coordinates and the optical pointer (a barcode reader, col. 21 lines 20-26). 

As to claim 5, Kuzunuki teaches the information input/output system according to claim 
1, wherein said environment type computer executes a processing operation of calling the linked 
digital object and/or displaying the digital object to the user in response to a user operation 



Application/Control Number: 09/739,952 Page 4 

Art Unit: 2179 

applied to the site of forming the link information on the surface of each of the physical objects 
on the basis of the image picked up by said imaging means (e.g., col. 3 lines 44-52 and col. 19 
lines 41-65). 

As to claims 6-7, they are similar in scope to claim 1 above; therefore, rejected under 
similar rationale. 

As to claim 9, Kuzunuki teaches an information input/output system to be used for user 
operations relating to an object in an information space realized by expanding a digital space of a 
computer into the real world, said system comprising: 

one or more than one operation surfaces arranged in said information space; a display 
means for displaying an image on said operation surfaces; an imaging means for picking up an 
image of said operation surfaces; a stationary environment type computer arranged in said 
information space; and one or more than one physical objects mounted on said operation surfaces 
and containing a visually identifiable visual marker on the surface thereof (note the rejection of 
claim 1 above); 

at least one of said physical objects being a virtual camera having a virtual imaging 
direction as specified by the position information contained in the visual marker (e.g., the 
facsimile direction to be placed v^ithin a given distance, col. 2 line 66-col. 3 line 11). 

As to claim 10, Kuzunuki teaches the information input/output system according to claim 
9, wherein 

said environmental type computer is adapted to execute (e.g., col. 19 lines 41-65); 
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a processing operation of recognizing/identifying said virtual camera and identifying the 
position information and the virtual imaging direction (a video camera 6 mounted above a desk 
surface 2 for picking up physical documents, col. 6 lines 1-7, and fig. 1); 

a processing operation of generating a virtual picked up image according to the position 
information and the virtual imaging direction (e.g., col. 2 line 66-col. 3 line 11, and col. 10 line 
56-col. 1 1 line 27); and 

a processing operation of displaying the virtual picked up image to the user (figs. 6A-8B). 

As to claim 12, Kuzunuki teaches an information input/output system to be used for user 
operations relating to an object in an information space realized by expanding a digital space of a 
computer into the real world, said system comprising: 

one or more than one operation surfaces arranged in said information space; a display 
means for displaying an image on said operation surfaces; an imaging means for picking up an 
image of said operation surfaces; a stationary environment type computer arranged in said 
information space; one or more than one physical objects mounted on said operation surfaces and 
containing a visually identifiable visual marker on the surface thereof (note the rejection of claim 
1 above); and 

an optical pointer adapted to point a specific position by irradiating a beam of light with a 
predetermined wavelength (using infrared light, col. 21 lines 20-26); 

said imaging means being capable of identifying the position pointed by said optical 
pointer (a barcode reader, col. 21 lines 20-26). 

As to claim 13, Kuzunuki teaches the information input/output system according to claim 
1 2, wherein 
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said environment image type computer executes on the basis of the picked up by said 
imaging means (e.g., col. 19 lines 41-65); 

a processing operation of recognizing the user operation on said operation surfaces, using 
said optical pointer (a barcode reader, col. 21 lines 20-26); and 

a processing operation of controlling the display of the digital object by said display 
means according to the result of the recognition (e.g., col. 21 lines 20-58). 

As to claim 15, Kuzunuki teaches an information input/output system to be used for user 
operations relating to an object in an information space realized by expanding a digital space of a 
computer into the real world, said system comprising: 

one or more than one operation surfaces arranged in said information pace; a display 
means for displaying an image on said operation surfaces; an imaging means for picking up an 
image of said operation surfaces; a stationary environment type computer arranged in said 
information space; one or more than one portable computers capable of exchanging digital 
objects with other computers; one or more than one physical objects mounted on said operation 
surfaces and containing a visually identifiable visual marker on the surface thereof (note the 
rejection of claim 12 above); and 

a portable ED recognition device adapted to recognize the visual marker on the surface of 
each of said physical objects and being capable of holding or releasing the recognized 
identification information (e.g., col. 21 lines 20-58). 

As to claim 16, Kuzunuki teaches the information input/output system according to claim 
15, wherein 

said environment type computer executes (e.g., col. 19 lines 41-65); 
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a processing operation of obtaining a digital object corresponding to the held 
identification information in response to the holding operation of said ID recognition device 
(e.g., col, 21 lines 20-58); and 

processing operation of transferring a digital object corresponding to the held 
identification information to a nearby physical object in response to the releasing operation of 
said ID recognition device (e.g., col. 21 lines 20-58). 

As to claim 22, it is individually similar in scope to claim 1 above; therefore, rejected 
under similar rationale. 

As to claim 24, it is individually similar in scope to claim 4 above; therefore, rejected 
under similar rationale. 

As to claim 25, Kuzunuki teaches the information input/output system according to claim 
22, further comprising: 

a step of calling the linked digital object and/or displaying the digital object to the user in 
response to a user operation applied to the site of forming the link information on the surface of 
each of the physical objects (e.g., col. 3 lines 44-52, link betv^een physical objects and 
electronically stored information, col. 19 lines 36-54). 

As to claim 26, it is individually similar in scope to claim 7 above; therefore, rejected 
under similar rationale. 

As to claim 27, it is individually similar in scope to claim 1 above; therefore, rejected 
under similar rationale. 

As to claim 28, it is individually similar in scope to claim 13 above; therefore, rejected 
under similar rationale. 
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As to claim 30, it is individually similar in scope to claim 16 above; therefore, rejected 
under similar rationale. 

Claim Rejections - 35 USC § 103 
5. Claim 2, 3, 8, 11, 17-21, 23, and 31-35 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Kuzunuki et al. (U.S. Patent No. 5,917,490) in view of Pierre Wellner ("The 
Digital Desk Calculator," Proceedings of ACM Symposium on User Interface Software and 
Technology (UIST '91), 11-13 Nov. 1991), 

As to claim 2, Kuzunuki teaches the information input/output system according to claim 
1, wherein at least one of said physical objects is capable of being moved in said information 
space and exchanging digital objects with others (note the rejection of claim 1 above); however, 
Kuzunuki does not teach that the physical object is a portable computer capable of being moved 
in said information space and exchanging digital objects with other computers. An electronic 
object (physical object) of Wellner clearly discloses concept of controlling/interacting with the 
electronic object by scanning, and then projecting it back on an operating interface (see page 30 
column 1-2). Moreover, Wellner's invention is not only limited that the electronic object is a 
DigitalDesk Calculator but also meant the same concept of using the electronic object as 
mentioned above can be applied on any regular PC if it gets scanned and projected on the 
operating interface ("the DigitalDesk Calculator is an example that illustrates how merging these 
two worlds can improve the usability of a very simple and well-established application," sic, see 
Conclusion on page 32). It would have been obvious at the time of the invention that a person 
with ordinary skill in the art would want to have the system allows the user to interact with the 
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electrical object of Wellner in the environment of Kuzunuki to share computer functionality to 
the physical objects and environment while allov^ing users to continue interacting v^ith the 
objects as before (column 1, page 32). 

As to claims 3 and 23, they are similar in scope to claim 2 above; therefore, rejected 
under similar rationale. 

As to claim 8, it is similar in scope to claim 2 above; therefore, rejected under similar 
rationale. 

As to claim 1 1, it is similar in scope to claim 2 above; therefore, rejected under similar 
rationale. 

As to claim 17, Kuzunuki in view of Wellner teaches an information input/output system 
to be used for user operations relating to an object in an information space realized by expanding 
a digital space of a computer into the real world, said system comprising: 

one or more than one operation surfaces arranged in said information space; a display 
means for displaying an image on said operation surfaces; an imaging means for picking up an 
image of said operation surfaces; a stationary environment type computer arranged in said 
information space; one or more than one portable computers capable of exchanging digital 
objects with other computers; and one or more than one physical objects mounted on said 
operation surfaces and containing a visually identifiable visual marker on the surface thereof 
(note the rejections of claims 2 and 12 above); 

said environment type computer being adapted to allow mouse/cursor operations on said 
portable computers to extend to said operation surfaces and display a rubber band as visual 
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feedback (actual object handling, col. 9 lines 6-29) according to the area on said operation 
surfaces as defined by means of said mouse/cursor (note the rejection of claim 2 above). 

As to claim 18, it is similar in scope to claim 17 above; therefore, rejected under similar 
rationale. 

As to claim 19, Kuzunuki teaches the information input/output system according to claim 
17, wherein another object can apply a process or a method it possesses to the rubber-banded 
original digital object in response to an action of said original object of being dropped on said 
another object (actual object handling, col, 9 lines 6-29, and figs. 6A-8B). 

As to claim 20, Kuzunuki teaches the information input/output system according to claim 
19, wherein said another object is a physical object possessing a process or a method for printing 
a document (printers, col. 15 hnes 1-10). 

As to claim 21, Kuzunuki teaches the information input/output system according to claim 
19, wherein said another object is a name card possessing a process or a method for mailing a 
document to the mail address corresponding to the name on it (e.g., col. 8 lines 15-31, and fig. 
3A). 

As to claim 31, it is individually similar in scope to claim 17 above; therefore, rejected 
under similar rationale. 

As to claim 32, it is individually similar in scope to claim 18 above; therefore, rejected 
under similar rationale. 

As to claim 33, it is individually similar in scope to claim 19 above; therefore, rejected 
under similar rationale. 
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As to claim 34, it is individually similar in scope to claim 20 above; therefore, rejected 
under similar rationale. 

As to claim 35, it is individually similar in scope to claim 21 above; therefore, rejected 
under similar rationale. 

Response to Arguments 

6. Applicant's arguments filed in the communication have been fiilly considered but they 
are not persuasive. 

Applicants have argued and Examiner does not agree with the following reasons: 

a. Kuzunuki does not teach or suggest moving image on a display screen such as a 
computer screen from the computer screen to a physical object such as a tabletop or a 
wall of room. 

Kuzunuki clearly teaches the image monitored by overhead camera 300 is fetched 
into image input I/F 406, then stored in memory 402. Data in memory 402 is processed 
by CPU 401 and the result is output from projectors 105 and 107 by image output I/Fl 
408 or image output I/F2 409 as an image to prove that the actual image can be 
picked/selected by the user and then projected back on the different location on the screen 
by using the projector (e.g., col. 7 lines 31-35). 

b. Kuzunuki does not teach a processing operation of recognizing the digital object 
dropped to a site on a surface of each of the physical objects and a processing operation 
of forming link information linking the digital object to the drop site on the surface of 
each of the physical objects. 
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Kuzunuki clearly teaches the environment in which an image object can be 
handled in a sense of direct touch and the environment in which actual objects can be 
used as man-machine interface parts are established (e.g., col. 7 lines 43-57), and 
transferring of the object to be displayed by a facsimile is explained in FIG. 8. Also in 
FIG, 8 (a), the facsimile transfer operation is judged as the one-handed operation as in 
FIG. 6. Here, the facsimile transmission destination is assumed to have been defined in 
advance. As in FIG. 7, when an image object is moved to facsimile part 1 12, image 
object 104 is highlighted to notify the operator of the reception (sic, col. 10 lines 32-44). 
It clearly shows that the digital object can be dropped to the other interface and having 
the similar functions as the original. 

c. Kuzunuki does not teach the physical object is a portable computer capable of 
being moved in the information space in exchanging digital objects with other computers. 

Kuzunuki in view of Pierre Wellner clearly teaches (claim 2) the electronic object 
(physical object) of Wellner clearly discloses concept of controlling/interacting with the 
electronic object by scanning, and then projecting it back on an operating interface (see 
page 30 column 1-2). Moreover, Wellner's invention is not only limited that the 
electronic object is a DigitalDesk Calculator but also meant the same concept of using the 
electronic object as mentioned above can be applied on any regular PC if it gets scanned 
and projected on the operating interface ("the DigitalDesk Calculator is an example that 
illustrates how merging these two worlds can improve the usability of a very simple and 
well-established application," sic, see Conclusion on page 32). It would have been 
obvious at the time of the invention that a person with ordinary skill in the art would want 
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to have the system allows the user to interact with the electrical object of Wellner in the 
environment of Kuzunuki to share computer fiinctionality to the physical objects and 
environment while allowing users to continue interacting with the objects as before 
(column 1, page 32). 

Conclusion 

1, THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1 .136(a), 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS fi*om the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1 .136(a) will be calculated fi-om the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the mailing 
date of this final action. 

8. Any inquiry concerning this communication or earlier communications fi-om the 
examiner should be directed to True T Chuong whose telephone number is 571-272-4134. The 
examiner can normally be reached on M-Th and alternate Fridays 8:30 AM - 5:00 PM. 

If attempts to reach the examiner by telephone are unsuccessfijl, the examiner's 
supervisor. Heather R. Hemdon can be reached on 571-272-4136. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 
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Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 



True T. Chuong 
03/01/05 



